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ABSTRACT 

It is argued that the three basic issues that have 
typically been the concern of formative evaluation 

(comprehensibility, appeal, and usefulness) tell us very little about 
the role that an instructional product can play in the ongoing 
environment of the classroom. The focus in this paper is on the field 
test, i.e., on research situated within the framework of existing 
classroom activity. Four issues considered central to situated 
research are discussed: (1) How is the instructional prototype 
interpreted by teachers? (2) How does the prototype function as part 
cf the ongoing content curriculum tyat is both the backbone and 
blueprint for classroom goals and activities? (3) How does t.he 
prototype function within the organizational context of the 
classroom? and (4) Within the social context of the classroom, how is 
the prototype interpreted by students? Each of these questions is 
addressed using examples from formative research with tv;o different 
prototype products: a computer-based set of interactive cognitive 
tools to support inquiry practices in science ("Inquire")/ and a 
tele/ision series that uses a narrative forrat to introduce students 
to different arenas of scientific and mathematical inquiry ("The 
Second Voyage of the Mimi") . (8 references) (GL) 
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In the development of educational technologies, 
formative research has been concerned with 
addressing three basic issues: (1) comprehensi- 
bility, (2) appeal, and (3) usefulness (Honey, 
1987). Comprehensibility research investigates 
both the clarity of the educational ideas contained 
within the environment and, in the case of informa- 
tion technologies, the clarity of theinterface that enables 
the user to interact with the program. Appeal research 
addresses the popularity of the basic activity among 
the target audience, and usefulness research evalu- 
ates the educational effectiveness of the product. Al- 
though these three elements constitute the objectives 
of formative research, they tell us very little about the 
actual role that the product can play in the ongoing 
environment of the classroom. 

This paper focuses on a different and, we believe, 
an increasingly inrportant category of formative re- 
search for interactive technologies in education— the 
field test, that is, research situated within the frame- 
work of existing classroom activity. Due to the time 
and resource constraints inherent in almost all devel- 
opment projects, researchers often have no choice but 
to examine apart from the classroom setting how stu- 
dents understand, are excited by, and learn from 
educational technologies. However, if the product is 
intended for use in a classroom setting, then it be- 
comes imperative to examine how it functions within 
the complex environment of the classroom. 

This belief derives from a framework which as- 
serts that interactive technologies can play an impor- 

A version of this paper was presented at the American 
Educational Research Association in the symposium 'The Role of 
Formative Research as a Bridge Betxveen Developers and Software 
Users" San Francisco, CA, March 28, 1989. 



tant role in qualitatively improving learning and 
teaching only if these products are deeply integrated 
into theleamingsetting(Hawkins, Brunner,Ghitman, 
Mann, Magzanien, & Moeller, 1 988; Honey, Martin, & 
Robinson, 1987; Rubin & Bruce, 1986). We have in 
mind products that do not take over or replace the 
curriculum, but are truly integrated into and enhance 
an array of possible learning activities. Such products 
make use of the power of interactive technologies to 
extend what it is possible to do, and thus can refash- 
ion the classroom situation (Lampert, 1985; Martin, 
1987). To do this, formative information about how 
prototype products function in these social and intel- 
lectual contexts is needed. 

In the initial phases of formative research, the 
testing is generally carried out with individual chil- 
dren or with small groups of childrcii outside the 
classroom environment. This phase of the formative 
process is necessary to get a fine-grained reading of 
what features of the product are educationally effec- 
tive and appealing to students, and what aspects of 
the product facilitate or deter students' mteractions 
with it. This phase of research can also be used to 
solicit interview-type information from other rele- 
vant participants such as teachers and educational 
consultantr While useful to the designers and devel- 
opers of e tional technology this formative infor- 
mation can oe at best only a very rough pass at how 
things will really work. Situated research serves a 
different purpose. It is important not simply as a final 
check to ''polisir the details of a product, but it is 
critical to a problem that many schools and research- 
ers are now defining: how to effectively integrate 
technologies into teaching and learning (e.g., Collins 
& Hawkins, 1989; Martin, 1988). 

Situated research allows us to address different 
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kinds of formative questions from those explored 
during the early phases of the development process. 
While it can expand upon some of the fine-grained 
inquiries about exactly what students learn from the 
experience (e.g., how does the process of using a piece 
of ma thematici, software relate to individual students' 
development of insight into place-value), without a 
large budget and long timeline, this is often not the 
best context to address those categories of formative 
questions. Situated research, however, can address 
questions concerning how the educational technol- 
ogy functions within the complex environment of the 
classroom. More specifically, what are the occasions 
of interpretation and use that enable students and 
teachers to engage with educational technology in a 
productive way? 

Within the framework of this paper we will dis- 
cuss four issues that are central to situated research: 

1 . How is the prototype interpreted by teachers? 

2. How does the prototype function as part of the 
ongoing content curriculum that is both the backbone 
of and the blueprint for classroom goals and activi- 
ties? 

3. How does the prototype function within the 
organizational context of the classroom? 

4. Within thesocial context of theclassroom, how 
is the prototype interpreted by students? 

These questions address not just the issue of how 
aproduct/i'fs imo existing circumstances, but issues of 
potential change made possible by patterns of appro- 
priation and use employed by teachers and students 
in relation to the prototype. Such identifiable patterns 
maybe useful in facilitating our understanding of the 
ways in which different educational contexts can be 
modified to support creative and lasting practices for 
integrating technology in a deep and meaningful 
way. Research methods such as preliminary discus- 
sions with teachers, formal and informal interviews 
with teachers and students conducted throughout the 
research process, and observations of how teachers 
and students make use of the prototype materials are 
useful tools in examining the prototype in a situated 
research context. 

We will illustrate each of the above questions by 
using examples from formative research we have 
done with two different prototype products: a com- 
puter-based set of interactive cognitive ''tools'' to sup- 
port inquiry practices in science; and a television 
series that uses a narrative format to introduce stu- 
dents to different arenas of scientific and mathemati- 



cal inquiry. The former prototype. Inquire, is illustra- 
tive of a case in which the developers and researchers 
had to make certain modifications in the software and 
the design of the field test. The later prototype. The 
Second Voyage of the Mimi^ is an example of situated 
research under the mostoptinial circumstances — there 
was an ideal match between the developers' goals and 
the teacher's. 

Inquire is a set of computer-based interactive 
cognitive tools to help middle-school students con- 
duct inquiry projects in science. The tools provide 
structure and support for processes of question for- 
mulation and revision, for information <^inding and 
recording, and for interpretation of qualitative and 
quantitative information to solve problems. In the 
field test, an eighth grade clas? in a New York City 
public school worked with Inquire over a two-month 
period as part of their required science-project work. 
Students investigated a set of problems in sports 
physics, and were given a wide variety of materials in 
multiple media (video, computer simulations, and 
laboratory programs, as well as a variety of text re- 
sources and a human expert) as some of the source 
material for their investigations. 

The Second Voyage oftheMimi is a federally funded 
multimedia project in science and math education for 
upper-elementary school students. The centerpiece of 
the Mi'mi is a 24-part television show that focuses on 
Maya archeology in the Yucatan Peninsula of Mexico. 
Th^ story features a young boy and his grandfather 
who, through a series of coincidences, end up in- 
volved in an adventure that leads them into an exten- 
sive exploration of Maya life—both past and present. 
The situated research that we conducted with the 
video dramas took place over the course of several 
months in a fifth/sixth grade classroom in an alterna- 
tive elementary school in East Hariem. 

1. Hoiu Is the Prototype Interpreted by Teachers? In 
ijeneral terms, teachers interpret technological proto- 
types in relation to the following kinds of concerns: 
tneir educational goals and objectives; the fitbetween 
a particular problem situation and the technology; 
and their orientation to the value and function of 
technology within a classroom setting. Within the 
fran^eworlc of situated research, it is the responsibility 
of researchers to take the concerns of teachers seri- 
ously. The result is that teachers become collabora- 
tors, giving shape and focus to the design of the 
research. 
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In discussions with the teacher, the Inquire field 
test was designed to fit into the science re^arch 
project work that was required of students in the 
spring semester. The developers' goals were to see 
how the procedures supported students' thinking 
and their use of a variety of content materials. Ihe 
teacher's primary goals, however, were slightly dif- 
ferent. He was more interested in how the technology 
could provide an occasion for talking about and prac- 
ticing the social skills involved in small-group, col- 
laborative work on a science project. He wanted to use 
the system to make very explicit the processes groups 
must engage in (e.g., brainstorming, how to exchange 
information if students work on different parts of the 
problem), and to cons-riously demonstrate and prac- 
tice thesegroupprocessskills. He, skeptically, wanted 
to see if the technology could play a substantial role in 
addressing this difficult problem. The design of the 
field test was altered somewhat to accommodate this 
goal, and the results indicate that the interactive pro- 
cedures made explicit by the tools successfully en- 
gaged students in a new way in small-group investi- 
gations. 

With The Second Voyage of the Mimi materials, we 
were invited by the teacher to use his classroom as a 
site for formative evaluation. His students were in 
their firsc month of what was to be a year-long inves- 
tigation of Maya culture. He saw the dramatic videos 
as a way to supplement and enhance an investigation 
that was already under way. He chose to use the 
videos as a jumping-off point with which to corrobo- 
rate and expand upon issues already under discus- 
sion. As researchers, we were able to accumulate a 
wealth of information on the creative use of these 
materials and to incorporate this information into the 
design of the accompanying print materials. 

2, HoiP Does the Prototype Function as Part of the 
Ongoing Curriculum? If the technology does not con- 
nect substantively with the ongoing work of the class- 
room, then it will always be peripheral, regardless of 
the good intentions of the developers. Ideally, the 
product should be flexible enough so that it can be 
used in a variety of classroom settings. Previous re- 
search and development must have been "good 
enough" to ensure that the product can accommodate 
a wide range of anticipated users. Using the context of 
situated research^ the evaluator, in collaboration with 
the teacher and the students, must help to disco ver the 
conditions for that substantive connection. 



Since Inquire is a set of content-free tools, the trick 
was to see how it could become integrated into the 
ongoing science curriculum — in this case "energy." 
The developers were dedicated to the notion that 
student inquiry work should grow, at least in part, 
from their own interests— no easy task when the focal 
content area for the development work was physics, 
and v/e wanted to substantially involvegiris. Through 
previous formative research, we found that the topic 
of sports physics was quite motivating to students of 
this age, including a comfortable number of gir's. 
Accordingly, the teacher chose to highlight this as the 
area for studencs' science projects. The development 
team found and developed a range of multimedia 
materials for students to work with—new to this 
classroo m— and the teacher chose consciously to build 
bridges between the physical science information he 
formally presented and students' project work. We 
developed a mode of working where we discovered 
^uite concretely what needed to be negotiated to 
integrate these tools into the content curriculum. The 
study provided a great deal of information about the 
kinds of content materials for student projects that 
work in conjunction with more formal curriculum 
presentations and that might be included in "content" 
disks that accompany the general inquiry tools. 

The situated research conducted with the Mimi 
materials made it all the more apparent that students' 
understanding of the material contained in the video 
was greatly enhanced when their viewing was sup- 
plemented with ongoing projects and additional in- 
vestigations. As part of their course of study, students 
were required to carry out projects on particular as- 
pects of Maya culture that captured their imagination. 
Students chose a variety of activities, including learn- 
ing to read Ma> a hieroglyphs and making a quilt that 
used the glyphs to tell a story; canying c at ai; inves- 
tigation of the Maya number system; building Maya 
temples; making a clay map of the region the Maya 
lived in; and writing stories about the ancient Maya. 
At the end of the year, the students wrote and pro- 
duced a class play, called "Vacation with the Maya," 
that dealt with the worid of both the contemporary 
and ancient Maya as seen through the eyes of two 
young students from East Harlem. 

3. How Does the Prototype. Function rvithin the 
Organizational Context of the Classroom? Within 
the social and intellectual context of a classroom, there 
is a complex stream of activity taking place thatdiffers 
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significantly from laboratory settings and meeting 
rooms where initial formative testing is frequently 
done. The introduction of educational technologies 
mto thedassroom set tingcan potentially impact upon 
as well as be affected by the organizational environ- 
ment. On the one hand, researchers may find that the 
prototype is used in ways they had not anticipated; on 
the other, researchers may observe that the prototype 
supports organizational practices that had not previ- 
ously been in use 'Martin, 1987). This information 
becomes important in the design of print materials 
that can support the creation cf different learning 
contexts. 

A? discussed above, the teacher interpreted the 
primary pov^er of Inquire to be explicit procedural 
support for i>mall-group work. Through the field test, 
he found that his role changed in relation to this work 
organization. Before the software, he found that he 
spent much of his time explaining inquiry procedures 
to students, reminding them to take notes, and so 
forth. The software took over this low-level work by 
simply providing the activity framework to which 
students responded. This allowed the teacher, in his 
words, to "up the ante" of his involvem.ent with 
students. He now had the time to give the "metacog- 
nitive" support necessary to complex problem solv- 
ing: to remind individual students of the big picture of 
the problem when needed, to help them inlerpret 
their information, and to arrange for groups produc- 
tively to exchange information at appropriate junc- 
tures. 

The class using theMfmz materials had in place a 
practice where they gathered, "town meeting style/' 
forapproximatelyonehoureachday. This format was 
used as an occasion to work on group process. Stu- 
dents were encouraged to listen to each other and 
understand that individuals'interpretations may vary 
dependingupontheirparticularcultural backgrounds. 
The Mimi materials were incorporated into this ongo- 
ing activity. For several months, the town meetings 
centered around the video materials, and discussion 
and debate took place after the airing of each drama. 
BecaasetheMzmf materials are about exploring,inter- 
preting, and understanding another culture, the 
teacher was able to use the videos as an occasion to 
support and enhance a process that was already in 
place. 

4. Within the Social Context of the Classroom, How 
Is the Prototype htterpreted by Students? In the typi- 
cal formative setting, students frequently engage wi th 



prototypes out of the classroom context. They are 
often asked to interact with the materials as isolated 
individuals or as isolated pairs — not as members of a 
preexisting group. When a prototype is introduced 
into a classroom setting, students will interpret it 
depending upon what they think its purpose is in 
relation to what "counts" wifhin the school environ- 
ment. Thus, the way ir^ which the prototype meshes 
with the t'ominant liscouroe of the classroom can 
make all t'le difference in how students view the ma- 
terials. 

In the Inquire classroom, issues of the legitimacy 
of this work arose pretty quickly — who were these 
people and why was all this discussion of group work 
processes going on? ThL* larger framework of their 
required science projects helped students to see the 
place of the software in classroom activity and how it 
might actually be useful to them; they also really liked 
the fact that computers were now central to their 
science work. However, students struggled with the 
novel nature of these science projects they weie being 
asked to do; some viewed this kind of information- 
finding-and-integrating inquiry like the research they 
did in history and nothing like the science projects 
they had done inthe past. The foregroundingof group 
work processes was also a new and questionable 
activity to students, and the teacher found extra ses- 
sions to be necessary to help students understand 
how all the new pieces fit together. Studen ts' interpre- 
tation or the field-test activity was essential to inves- 
tigate because, in creating a print context and intro- 
ducing ^he software, successful integration requires 
an understanding of how students define legitimate 
science project work — where do they put their ef- 
forts? 

For the students in the class in East Har'em, the 
Mimi dramas became an important part of their class- 
room culture. On the one hand, the materials were 
highly effective in bringing to life the students' inves- 
tigations of Maya life. On the other, the characters in 
the dramas provided an occasion in which meaning- 
ful identifications could take place and, as a result, 
students became invested in the struggles and con- 
cerns of the protagonists. In this respect, the connec- 
tion to \heMimi materials was both intellectual and 
personal and, ultimately, much more meaningful to 
the students. 

Several conclusions can be drawn that summarize the 
importance of conducting research within the ongo- 
ing framework of the classroom. 
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First, because situated research is carried out at 
the end of the development process, the prototype 
must be robust enough to funcrion withoat too many 
problems; it cannot be substantially revised at this late 
phase of development. Hov/ever, minor changes can 
be made that can significantly affect the ways in which 
students interact with the product and/or under- 
stand it. In the case of Inquire, we were, for example, 
able to insert procedures that allowed kids to identify 
their ov/n work when working with others in a small- 
group context. In the case of the Mimi dramas, we 
were able to inform the production staff that the 
episode which contained a lesson on Maya math was 
enormously confusing to kids; as a result the produc- 
ers re-shot that segment. 

Second, the information collected in a situated 
research setting can have an enormous impact on the 
design and development of the accompanying mate- 
rials. With the Inquire program, we were interested in 
developing content disks, which would accompany 
the general tool. The work that the students did in 
sports physics provided us with a wealth of relevant 
^'nformation about the kinds of materials that work in 
that and, by implication, other domains. In the case of 
the Mimi research, at the conclusion of the academic 
year the teacher was hired by the development staff to 
write the teacher's guide for the video dramas . He was 
able to effectively incorporate into the guide the infor- 
mation that he had accumulated in thecoui-seof r Jing 
the materials. 

Third, if we have as our overriding goal the deep 
and meaningful integration of information technolo- 
gies into the framework of existing classroom activi- 
ties, then it is essential to understand the multiple 
ways in which this can happen. As researchers we can 
document tl'e factors that lead to both successful and 
unsuccessful experiences in situated research settings. 
We can use the information collected in a variety of 
classroom contexts to help us identify the vanables 
that lead to successful models for technology integra- 
tion. In this respect, classroom formative tests become 
a source of interesting information that can be accu- 
mulated over studies and applied as a body of knowl- 
edge to new product development. 

If we are to take seriously the task of understanding 
how technology can be integrated into classroom 
settings, we need a flexible approach that can accom- 
modate the nuances of various classroom enviro^v- 
ments. Most important, we must listen to the needs of 



teache/s and students and enlist their support as 
collaborators and shapers of the research process. 



Nofres 

The work reported here on Inquire was supported by 
Grant # NSFMDR-8550399 from the National ScienceFoun- 
dation programin Applications of Advanced Technologies. 
The work reported on The Second Voyage of the Mimi was 
funded jointly by the U.S. Department of Education and the 
National Science Foundation, with funding administered 
by the D.O.E. under Grant # G008510039 The Mfmf materi- 
als are published by Sunburst Communications. Samuel Y. 
Gibbon,Jr.istheexecutiveproducerofBank Street's Project 
iA Science and Mathematics, which produces the Mimi 
materials. 



References 

Collins, A., & HawkinsJ. (1989). Introducing the Center for 
Technology in Education. Paper presented at the annual 
meeting of the American HUacational Research Asso- 
ciation, San Franciscv;, CA. 

Hawkins,J.,Brunner,C,Ghitman,M.,Mann,F., Magzamen, 
S., & Moeller, B. (1988). Inquire in classrooms: A cognitive 
tool-based approach to leamin^Jn science. Paper presented 
at the annual meeting of the American Educational 
Research Association, New Orleans, LA. 

Honey, M. (1987). The role offo/mative research in tliedesign of 
educational computer software. Paper presented at the 
American Evaluation Association, Boston, MA. 

Honey, M.,Martin, L, & Robinson,S. (1987, October). Teach- 
ing technology: Creating environments for change. 
SIGTE Bulletin. 

L?mpen, M. (1985). Mathematics learning in context: The 
Voyage of the Mimi. Journal cf Mathanaiical Behavior. 
4(2), 157-168. 

Martin, L. (1987). Teachers' adoption of multimedia tech- 
nologies for science and mathematics instruction (pp. 
35-56). In R. D. Pea iL- K. Sheingold (Eds.), MirroTS of 
minds: Patterns o/ experience in educational computing. 
Ablex: Norwood, NJ. 

Martin, L. (1988). System-wide factors in sustaining a technol- 
ogy-based inquiry environment. Paper presented at the 
annual meeting of the American Educational Research 
Association, New Orleans, LA. 

Rubin, A., & Bruce, B. (1986). Leaning with QUILL: Lessons 
for students, teachers and '"oftware designers. In T. E. 
Raphael (Ed .), Contents of school-based literacy . New York: 
Random House. 



ERLC 



7 



